Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add iris-grib to dev dependencies #980

Merged
merged 2 commits into from
Mar 4, 2025
Merged

Add iris-grib to dev dependencies #980

merged 2 commits into from
Mar 4, 2025

Conversation

jfrost-mo
Copy link
Member

@jfrost-mo jfrost-mo commented Dec 10, 2024

It seems that the eccodes download issues were just temporary, so we can now re-add it.

Fixes #959

Contribution checklist

Aim to have all relevant checks ticked off before merging. See the developer's guide for more detail.

  • Documentation has been updated to reflect change.
  • New code has tests, and affected old tests have been updated.
  • All tests and CI checks pass.
  • Ensured the pull request title is descriptive.
  • Conda lock files have been updated if dependencies have changed.
  • Attributed any Generative AI, such as GitHub Copilot, used in this PR.
  • Marked the PR as ready to review.

@jfrost-mo jfrost-mo added the dependencies Pull requests that update a dependency file label Dec 10, 2024
@jfrost-mo jfrost-mo self-assigned this Dec 10, 2024
Copy link
Contributor

github-actions bot commented Dec 10, 2024

Coverage

@jfrost-mo
Copy link
Member Author

jfrost-mo commented Dec 10, 2024

Okay, maybe not. But it worked when I tried it manually?

The difference here is that when resolving manually it went via artifactory, while building from the lock files goes to anaconda.org.

But then how did it work from Firefox.

It works on GitHub Actions resolving from the same lockfiles, so unless GitHub have overwritten the DNS location of anaconda.org, we should be trying the same request.

Other .conda files can be downloaded from anaconda.org, including eccodes releases. Eccodes does seem particularity slow however. I suspect it is exceeding the 60 second timeout.

Theory

Something is funky about the eccodes package on anaconda.org, causing the anaconda.org server to have to do a bunch of work to retrieve it. This work takes longer than 60 seconds, so the HTTP timeout on conda times out, and the package fails to download. Artificatory is not affected as it will have its own backing store.

Actions

  • Try installing with increased timeout (how?)
  • Investigate what is different with eccodes. The CSET package is much faster to start downloading.
  • If we discover something, report upstream to conda-forge.

@jfrost-mo
Copy link
Member Author

Installing with increased timeout

As best as I can tell the timeout is hardcoded. I guess we could hack our conda install just to test? Or see if mamba or pixi give more options.
https://github.com/conda/conda/blob/main/conda/base/context.py#L353

@jfrost-mo
Copy link
Member Author

I think this is a web proxy issue, and I've opened INC000263814 to try and get some logs from there.

@jfrost-mo
Copy link
Member Author

To work around this issue the conda timeout can be increased:

conda config --set remote_read_timeout_secs 1000

@jwarner8
Copy link
Contributor

Looks like 2.40 has been released (https://anaconda.org/conda-forge/eccodes/files), so we can test this again to see if this works.

@jfrost-mo
Copy link
Member Author

jfrost-mo commented Feb 21, 2025

Testing again, conda still times out unfortunately. (Only two seconds over the threshold now though...) One can set the remote_read_timeout_secs to get it working, but it seems broadly the same as the last version.

If I had to guess, I would suspect there is some virus scanning process happening inside the web proxy that is getting stuck on this file, then timing out after a minute. This may not be entirely surprising, as the eccodes package contains about 25000 files.

@jfrost-mo
Copy link
Member Author

It works at least!

@jfrost-mo jfrost-mo marked this pull request as ready for review March 4, 2025 12:17
@jfrost-mo jfrost-mo requested a review from jwarner8 March 4, 2025 12:17
Copy link
Contributor

@jwarner8 jwarner8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, we'll keep an eye on it and make sure it doesn't cause problems again. It'll be great to have this in to support grib and ML models.

@jfrost-mo jfrost-mo merged commit 23b0c7c into main Mar 4, 2025
8 checks passed
@jfrost-mo jfrost-mo deleted the grib_support branch March 4, 2025 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dependencies Pull requests that update a dependency file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

GRIB support
2 participants